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Abstract 

The  training  of  a  surgeon  includes  the  acquisition  of  a  number  of  characteristics. 
These  include  a  cognitive  knowledge  base,  problem  formulation  and 
decisionmaking  abilities,  appropriate  psychosocial  relationships,  and  other 
attributes  that  can  be  measured  with  objective  testing,  such  as  national  board  or 
specialty  certifying  examinations.  Perhaps  most  critical  to  the  surgeon,  however, 
are  the  core  technical  skills  of  the  profession.  A  battery  of  sophisticated  devices  is 
being  created  to  teach  and  provide  objective  evaluations  of  the  trainee’s  technical 
abilities.  These  innovative  state-of-the-art  simulation  devices,  used  to  teach  basic 
skills  and  surgical  tasks  through  repetitive  proctored  challenges,  will  enable 
detection  and  analysis  of  surgical  errors  and  near  miss  incidents  without  risk  to 
patients.  As  with  flight  simulation,  near  miss  detection  capabilities  anticipate 
potential  errors  before  they  occur  and  prevent  resulting  complications.  In 
collaboration  with  a  number  of  otolaryngology  residency  programs,  we  have 
developed  a  comprehensive  training  curriculum,  based  on  complementary 
simulation  tools  and  several  perceptual  and  spatial  ability  test  instruments,  to 
provide  objective  metrics  for  assessing  the  technical  skills  of  trainees.  These  tools 
include  an  endoscopic  sinus  surgery  simulator  for  procedural  training,  and  the 
Minimally  Invasive  Surgical  Trainer- Virtual  Reality  (MIST-VR)  system  for  basic 
surgical  skills  training.  Technical  errors  are  identified,  quantified,  and  used  to 
monitor  surgical  performance,  after  which  the  metrics  are  used  for  outcome 
analysis,  for  the  purpose  of  improving  patient  safety.  Validated  metrics  include 
time-to-completion,  errors,  economy  of  motion,  and  psychomotor  tracking. 
Endoscopic  sinus  surgery  is  a  suitable  operation  substrate,  since  it  is  a  frequently 
performed  procedure  (more  than  300,000  times  annually  in  the  United  States.) 
and  carries  a  significant  risk  of  injury  to  the  delicate  neighboring  eye  and  brain 
structures.  The  endoscopic  sinus  surgery  simulator  (ES3)  is  an  advanced  surgical 
simulation  device  based  largely  on  jet  pilot  flight  simulation,  and  provides  the 
trainee  with  appropriate  levels  of  challenge  and  instruction.  ES3  validation  studies 
are  reported  here,  along  with  a  description  of  the  World  Wide  Web-based 
Intelligent  Information  Interface  (WI3)  simulation  database. 
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Introduction 

The  emphasis  on  patient  safety  in  the  operating  room  has  focused  historically 
on  the  numerous  supportive  functions  involving  devices,  medications,  staffing, 
and  administrative  procedures,  but  rarely  on  the  surgeon’s  technique  and 
performance.  A  surgeon’s  training  occurs  during  residency  and  is  traditionally  a 
subjective  process,  as  is  the  assessment  of  technique  and  the  evaluation  of  skill, 
dexterity,  and  insight.  Given  that  competence  and  safety  are  of  paramount 
importance,  the  objective  measure  of  a  resident’s  progress  is  a  critical  factor  and 
one  that  his  or  her  teachers  address  diligently. 

Otolaryngology  residents  training  in  endoscopic  sinus  surgery  (ESS)  must 
learn  the  anatomy  and  physiology  of  the  sinuses,  the  relevant  disease  processes, 
and  the  concepts  and  skills  of  surgery.  Cadaver  dissection  may  be  performed,  but 
most  training  is  conducted  in  the  operating  room  under  the  supervision  of  an 
experienced  surgeon.  As  training  is  dependent  upon  the  patients  of  the  attending 
surgeon,  it  is  imperative  that  the  operating  room  (OR)  environment  is  kept  as  risk¬ 
free  as  possible.  A  high-risk  training  environment,  similar  to  that  encountered  by 
a  commercial  pilot  flying  in  turbulent  weather  or  by  a  military  pilot  flying  over 
hostile  territory,  cannot  be  created  for  the  sake  of  resident  education.  Excessive 
intraoperative  bleeding  would  be  an  analogous  example  of  such  a  high-risk 
situation  in  surgery.  Even  with  the  use  of  cadavers  (which  don’t  bleed)  for 
training,  once  a  single  dissection  is  completed,  it  cannot  be  “erased”  for  the 
purpose  of  repeating  the  procedure,  since  the  anatomy  has  been  pennanently 
altered. 

Current  training  of  otolaryngology  residents  in  ESS  includes  videotape 
viewings,  cadaver  dissection  (where  available),  and  direct  observation  of 
procedures  performed  in  the  OR.  As  residents  progress  in  their  training,  they  are 
given  a  more  active  role  in  the  procedure  and,  in  their  final  year,  they  become  the 
major  participants.  For  the  most  part,  the  technical  acumen  required  for  basic  ESS 
procedures  is  not  achieved  until  the  latter  half  of  a  resident’s  training.  With  the 
increasing  complexity  of  endoscopic  dissection  equipment  and  the  ever-increasing 
demand  to  broaden  the  indications  for  ESS,  there  is  a  growing  need  to  make 
residents  more  familiar  with  the  technical  skills  of  ESS  at  an  earlier  stage  in  their 
training. 

Optimal  visualization  is  critical  to  such  endoscopic  procedures,  and  the 
surgeon’s  understanding  of  anatomical  relationships  must  be  unambiguous  and 
maintained  throughout  the  surgery,  even  as  significant  tissue  is  removed.  This 
anatomical  understanding  is  necessary  if  the  surgeon  is  to  navigate  the  patient’s 
sinus  passages  safely  and  avoid  complications.  Thus,  learning  the  complex 
anatomy  of  the  head  and  the  relevant  anatomical  relationships  is  essential  to  ESS, 
as  is  the  skillful  practice  of  surgical  maneuvers. 

Computer-generated  surgical  simulation  affords  the  ability  to  create  an 
environment  that  imitates  the  real  surgical  world  without  inherent  risk  to  the 
patient,  and  the  technology  serves  to  overcome  many  of  the  limitations  of  current 
training  approaches.  Procedures  can  be  repeated  until  proficiency  is  achieved, 
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techniques  can  be  refreshed  or  learned  anew,  the  variability  of  case-mix  (or  lack 
thereof)  can  be  overcome,  and  competency  can  be  assessed  by  certifying  boards 
or  hospital  accreditation  panels,  prior  to  taking  a  single  patient  to  the  operating 
room.1  Moreover,  new  devices  can  be  tested  in  a  simulated  environment,  and  new 
procedures  can  be  rehearsed  or  improved,  without  exposing  patients  to  potential 
risk  of  harm. 

Simulation  has  the  potential  to  improve  patient  safety  and  reduce  surgical 
errors  by  screening  would-be  surgeons  for  demonstrable  aptitude  and  providing 
initial  training  experience,  while  at  the  same  time  promoting  ongoing  training 
with  a  consistently  reproducible  process,  enabling  periodic  surgical  skill 
assessments,  and  cultivating  proficiency  through  rehearsals  of  complex,  patient- 
specific  procedures.2 

The  use  of  innovative  simulation  devices  that  train  surgical  tasks  and  basic 
skills  through  repetitive  proctored  challenges  may  lead  to  better  detection  and 
analysis  of  potential  surgical  errors.  Error  reduction  and  surgical  expertise  can  be 
achieved  through  the  implementation  of  a  dynamic,  simulation-based  curriculum 
in  which  objective  measurements  are  acquired  and  skills  are  assessed.  With 
exposure  to  such  a  curriculum,  surgical  trainees  may  be  able  to  achieve  a 
considerable  level  of  prepatient  expertise  and  provide  a  higher  quality  of  health 
care  during  their  residency  training  and  beyond. 


Role  of  a  surgical  simulation 

Virtual  reality  (VR)  simulators  based  on  interactive  computer  graphics  have 
had  a  marked  impact  on  the  skill  level  of  military  and  commercial  pilots,  and  may 
hold  similar  promise  for  the  medical  field.  Over  the  past  3  decades,  computer- 
assisted  devices  have  been  used  to  augment  the  education  and  training  of  surgical 
residents  in  several  fields  with  significant  success.3  8  VR  simulation  already  has 
played  an  introductory  role  in  training  residents  for  laparoscopic,  gastrointestinal, 
plastic,  ophthalmologic,  dermatologic,  and  some  laryngologic  procedures.7  14  VR 
simulation’s  efficacy  as  a  teaching  tool  is  clear,  but  whether  it  is  superior  to 
conventional  teaching  methods  has  yet  to  be  determined.9, 10, 13 

A  report  by  Seymour  and  associates15  has  begun  to  address  the  value  of 
surgical  simulation  training  by  demonstrating  that  VR  training  has  a  positive 
impact  on  resident  performance  in  the  OR,  and  potentially  on  safety  as  well. 
Similarly,  we  have  developed  an  integrated  training  environment  using 
psychomotor,  visuo-spatial,  and  proprioception  measurement  instruments,  along 
with  two  sophisticated  simulators,  to  maximize  skill  acquisition  and  to  gauge 
trainee  progress.  One  of  these  simulators  is  the  ES3  (Lockheed  Martin,  Akron, 
OH),  which  permits  trainees  to  progress  through  a  series  of  increasingly 
challenging  simulation  exercises  in  the  course  of  learning  a  complete  sinus 
surgery  procedure.  The  other  simulator  is  the  Minimally  Invasive  Surgical 
Trainer- Virtual  Reality  (MIST-VR,  Mentice  Corp.,  Gothenburg,  Sweden),  which 
was  developed  to  train  and  measure  hand-eye  coordination  and  maneuvers 
analogous  to  surgical  procedures  in  an  abstract  environment,  using  spheres,  cubes, 
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and  virtual  surgical  instruments.  Although  these  two  devices  are  the  state-of-the- 
art,  they  are  indeed  rudimentary  when  one  envisions  the  detail  and  complexity 
contained  that  an  ideal  surgical  environment  would  possess.  Nevertheless,  we 
(and  others)  are  finding  them  to  be  useful  training  tools  for  teaching  many 
surgical  skills. 


The  endoscopic  sinus  surgery  simulator 

The  ESS  training  simulator  was  developed  in  response  to  the  need  for  a 
simulator  to  help  train  novice  sinus  surgeons.16, 17  Computer  graphics  and  haptic 
(force)  feedback  are  used  in  the  ES3  to  create  a  virtual  surgical  environment,  and 
the  system  has  been  used  to  teach  otolaryngology  residents.  A  team  consisting 
of  Lockheed  Martin  (fonnerly  Loral),  the  Immersion  Corporation,  the  University 
of  Washington  HIT  Lab,  and  the  Ohio  Supercomputer  Center  built  the  ES3  with 
the  sponsorship  of  the  U.S.  Army  Medical  Command  and  the  Telemedicine  and 
Advanced  Technology  Research  Center  (TATRC)  from  1995  through  1998, 
furthering  the  state-of-the-art  in  surgical  simulation. 

The  ES3  is  composed  of  four  principal  components:  (1)  a  Silicon  Graphics 
(SGI)  computer,  which  serves  as  the  simulation  host  platform;  (2)  a  PC-based 
haptic  system  controller,  which  provides  the  requisite  high-rate  control  over  a 
physical  instrument  handle  associated  with  a  set  of  virtual  surgical  instruments; 

(3)  a  virtual  voice  recognition  instructor  PC,  which  responds  to  spoken  commands 
controlling  the  simulator;  and  (4)  an  electro-mechanical  platform,  which  houses  a 
physical  replica  of  an  endoscope,  the  mechanically  linked  surgical  tool  handle, 
and  an  external  head  anatomy  mannequin.  The  simulator  permits  the  user  to 
perform  endoscopic  sinus  surgery  on  a  virtual  patient,  using  a  wide  range  of 
simulated  surgical  instruments.  The  surgeon  trainee  explores  the  virtual  anatomy 
by  manipulating  the  simulated  endoscope  in  the  nose  of  the  mannequin,  just  as 
they  would  with  a  real  patient.  With  the  aid  of  the  virtual  instructor  PC,  the 
trainee  can  make  audible  requests  for  various  virtual  instruments  and  can  see  them 
appear  subsequently  on  the  monitor.  The  surgical  field  appears  true-to-life  and  all 
instruments,  apart  from  the  endoscope,  provide  haptic  feedback  through  the 
universal  instrument  handle.  This  enables  the  student  to  feel  tissue  resistance 
appropriate  to  the  tool  (i.e.,  cutting  resistance  for  scissors,  vibration  for  a 
debrider).19 

As  the  student  advances  through  the  simulator  training,  they  are  given  more 
complex  tasks  to  perform.  The  student  begins  training  in  an  abstract  “novice” 
environment,  rather  than  with  a  virtual  patient,  and  performs  tasks  designed  to 
familiarize  them  with  the  endoscope  and  the  available  array  of  surgical  tools. 
Students  learn  with  the  help  of  predefined  navigation  path  aids,  strategically 
placed  targets  for  injection,  and  detailed  dissection  clues.  This  novice-level 
training  improves  hand-eye  coordination  through  an  immersive  experience.  Each 
task  builds  upon  the  skills  acquired  in  the  previous  task,  and  all  of  the  tasks  relate 
directly  to  the  complete  ESS  procedure.  Upon  achieving  an  appropriate  score  at 
the  novice  level,  the  trainee  then  graduates  to  the  intermediate  level,  where  the 
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abstract  training  aids  appear  in  the  context  of  sinus  anatomy  and  replicate  the 
actual  surgical  environment.  Navigation  and  injection  aids  correspond  to  their 
novice  level  counterparts.  The  system  prompts  the  trainee  to  perform  the 
appropriate  dissection  tasks  while  interacting  with  labeled  anatomical  structures. 
The  surgical  field  appears  true-to-life  at  the  advanced  level,  and  the  student  is 
expected  to  perform  procedures  without  the  benefit  of  computer-generated  aids.19 
The  virtual  instructor  points  out  mistakes,  errors,  and  misses,  and  can  be  called 
upon  to  suggest  the  appropriate  tool  for  performing  a  particular  exercise. 

The  simulator  records  overall  and  task-specific  scores  for  each  student’s 
performance,  including  the  time  needed  to  complete  a  task  and  accuracy  relative 
to  optimal  performance.  A  penalty  is  imposed  when  excessive  time  is  needed  to 
complete  a  task,  or  for  hitting  “hazards,”  which  represent  the  anatomical  areas  the 
surgeon  should  take  care  to  avoid.  These  include  the  optic  nerve  and  the  lamina 
papyracea.  Points  are  deducted  from  the  trainee’s  score  if  one  or  more  hazards  are 
dissected. 


Metrics 

How  can  surgical  safety  be  measured  appropriately?  Complications  are  not  the 
sole  standard,  since  mechanisms  in  current  training  practices  largely  prevent  their 
occurrence  (primarily  via  intervention  by  the  attending  surgeon).  The  flight 
training  analogy  is  very  much  in  order  in  this  regard:  A  skilled  observer  assesses 
the  trainee’s  performance  through  briefing,  debriefing,  and  over-the-shoulder 
evaluation,  in  addition  to  the  use  of  predetermined  metrics,  objective  simulator 
responses,  and  using  the  trainee’s  own  observations  (analogous  to  the  dictated 
operative  note)."  ""  And  while  an  intentional  adverse  situation  cannot  be  induced 
in  a  patient  for  the  purpose  of  training  or  testing  the  skill  of  the  student,  a 
simulator  can  achieve  this  goal  with  complete  safety. 

A  conference  on  Metrics  for  Technical  Skills  was  convened  as  a  component  of 
this  research  project,  with  the  goal  of  deriving  objective  measures  for  the 
individual  components  of  an  endoscopic  sinus  operation.  Errors  in  ESS  were 
identified,  and  each  error  was  classified  according  to  a  taxonomy  of  error  types 
(i.e.,  technical,  cognitive,  or  combined).  The  list  was  approved  by  a  consensus  of 
the  conference  participants.  Quantifiable  measures  then  were  defined  for  each 
error.  Some  errors,  such  as  “wrong  tool  choice,”  were  assigned  simple  “yes”  or 
“no”  measures.  Other  errors,  such  as  “injury  to  the  lacrimal  duct”  or  “too  much 
scope  rotation”  have  clear  and  measurable  end  points.  Other  errors,  such  as  “past 
pointing”  or  “lack  of  progress,”  may  need  further  quantification,  and  an  expert 
rater  panel  is  currently  evaluating  them  with  the  assistance  of  recorded 
procedures.  A  set  of  outcome  measures  was  developed  with  the  consensus  of  the 
conference  participants,  to  address  the  broad  areas  of  technical,  cognitive,  and 
combined  skills,  as  well  as  global  assessment  measures  of  surgical  proficiency. 
This  rigorous  process  has  brought  to  the  project  an  added  emphasis  on  error 
reduction  prevention  as  a  fonnal  component  of  surgical  training.  Students  are 
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taught  in  a  uniform  and  comprehensive  manner  precisely  what  the  errors  are  and 
how  to  avoid  them.2  ' 

The  key  to  error  prevention  is  not  so  much  the  detection  of  errors  and  related 
complications,  but  rather  the  detection  of  near  misses  and  the  prevention  of 
related  errors  and  complications.  One  of  the  strengths  of  the  aviation  safety 
industry’s  record  of  perfonnance  is  the  reporting  of  near  misses,  which  are 
stressed  in  simulation  training.  Because  the  ES3  can  accurately  measure  the  path 
of  the  virtual  instrument  and  how  close  to  an  anatomical  hazard  an  instrument  is 
placed,  it  can  measure  near  misses  that  may  or  may  not  result  in  a  postoperative 
complication.  Thus,  the  ES3  can  quantify  potential  errors  in  an  objective  manner, 
and  can  better  train  residents  to  avoid  them  by  providing  near  miss  detection  and 
feedback. 


Current  ESS  training 

We  conducted  a  survey  of  the  five  major  centers  participating  in  this  research 
project  in  an  effort  to  better  understand  the  present  state  of  resident  training.  The 
survey  asked  the  program  directors  19  questions  regarding  various  aspects  of 
teaching  ESS  to  residents.  These  include  questions  such  as  what  year  residents 
began  performing  ESS;  if  formal  lectures  are  required  prior  to  perfonning 
surgery;  what  is  the  curriculum  for  residents  reading  prior  to  surgery;  if  cadaver 
dissection  is  performed  and  if  skills  on  cadaver  dissection  is  evaluated.  Feedback 
to  residents  was  also  queried.  The  survey  yielded  the  following  results:  (1) 
residents  often  did  not  begin  performing  endoscopic  sinus  surgery  until  their  final 
year  of  otolaryngologic  training;  (2)  slightly  more  than  half  of  the  responding 
institutions  gave  formal  lectures  before  residents  were  allowed  to  perform  ESS, 
and  the  number  of  lecture  hours  varied  by  institution;  (3)  slightly  more  than  half 
of  the  institutions  required  the  completion  of  a  reading  list  before  residents  were 
allowed  to  perform  in  the  operating  room;  and  (4)  while  most  training  programs 
afforded  their  residents  the  opportunity  to  practice  their  ESS  techniques  on 
cadaver  heads  prior  to  their  initial  operating  room  experience,  none  of  the 
institutions  made  this  experience  a  requirement.  Based  upon  these  results,  we 
have  concluded  that  there  is  a  clear  need  for  an  ESS  curriculum  that  is  systematic, 
comprehensive,  and  well  structured,  and  includes  the  necessary  cognitive  content 
and  the  surgical  skills  training. 

A  comprehensive  and  integrated  curriculum  was  developed  to  standardize  pre¬ 
training  for  those  students  taking  part  in  the  ES3  simulation  studies.  This 
curriculum  was  compiled  on  a  compact  disc  (CD),  with  interactive  audio  and 
video  information  demonstrating  successful  and  erroneous  operations  at  each 
training  level,  as  well  as  written  background  and  anatomical  information. 
Preliminary  results  have  shown  that  this  curriculum  facilitates  to  a  great  extent  the 
trainees’  compliance  with  the  simulation  training  protocols  and  helps  to  eliminate 
bias  resulting  from  variations  in  the  educational  backgrounds  of  our  subjects. 
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ES3  validation 

Crucial  to  the  project’s  success  is  the  ability  to  validate  the  ES3  as  a  training 
tool  for  endoscopic  sinus  surgery.  One  of  the  studies  in  our  project  was  designed 
specifically  to  address  the  discriminant  and  concurrent  validity  of  the  ES3.  The 
purpose  of  this  study  was  two-fold:  (1)  it  compared  the  performance  levels  of 
medical  students  to  that  of  otolaryngology  residents  and  otolaryngology  attending 
physicians  over  23  trials  (10  at  the  novice  level,  10  at  the  intermediate  level,  and  3 
at  the  advanced  level);  and  (2)  it  established  benchmark  data  for  attending 
surgeons  that  can  be  used  to  measure  resident  and  medical  student  simulator 
performance. 

At  the  Montefiore  Medical  Center  (Bronx,  NY),  we  recruited  38  medical 
students  from  the  Albert  Einstein  College  of  Medicine  who  received  and  reviewed 
the  curriculum  materials  (as  discussed  above)  and  completed  a  demographic 
survey.  Research  technicians  then  gathered  performance  data  on  a  near-weekly 
basis  for  the  three  ES3  training  modes  (novice,  intermediate,  and  advanced). 
Performance  data  were  obtained  similarly  from  19  residents  and  17  attending 
physicians  at  the  Montefiore  and  Madigan  Army  Medical  Center  (Washington 
State)  simulator  sites.  New  attending  and  resident  data  is  being  added  on  an 
ongoing  basis. 

The  data  from  this  study  clearly  show  that  there  are  significant  group 
differences  in  ES3  simulation  training  performance  by  medical  students, 
residents,  and  attendings.  Otolaryngology  residents  performed  significantly  better 
than  the  medical  student  group  on  the  novice  and  intermediate  models  (Scheffe  F 
=  14.856,  P  <  .05,  and  Scheffe  F  =  6.236,  P  <  .05,  respectively).  Initial 
performance  on  the  novice  level  correlated  with  residency  training  year  and  prior 
ESS  experience  (r  =  .652,  P  <  .05).  In  support  of  the  simulator’s  face  validity, 
subject  ratings  of  the  virtual  model’s  realism  were  consistently  high  on  the 
postsession  questionnaire  and  in  open-ended  comments.  Postsession  questionnaire 
responses  also  confirm  that  the  otolaryngology  residents  perceived  the  simulator 
as  valid  and  useful  ESS  training  aid. 

As  one  would  expect  from  a  simulator  that  accurately  depicts  sinus  surgery, 
attending  physicians  performed  the  best  at  all  levels,  with  residents  close  behind. 
Although  the  medical  students  improved  their  performance  over  the  course  of  10 
trials,  their  performance  was  substantially  beneath  that  of  the  residents,  when 
using  the  performance  level  of  the  attendings  as  the  standard  (Figure  1).  Given  the 
numbers  of  participants  that  were  enrolled  from  all  of  the  sites,  the  results  bear 
substantial  statistical  power.  These  findings  are  unique  in  that  no  other  benchmark 
criteria  for  sinus  surgery  have  been  reported  previously. 

A  second  study  was  designed  to  establish  construct  validity  by  examining  the 
correlation  between  subject  performance  on  the  ES3  and  respective  performance 
on  other  validated,  objective  tests  of  fundamental  visuo-spatial  and  perceptual 
abilities. 24-28  Endoscopic  sinus  surgery  requires  simultaneous  two-handed 
coordination  of  surgical  instruments  and  an  endoscope  in  three-dimensional 
space.  Successful  performance  of  this  task  requires  complex  ambidextrous 
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Figure  1.  Mean  ES3  novice  level  scores  for  the  3  subject  groups  across  10  distributed 
practice  trials 
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(novice 
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psychomotor,  visuo-spatial,  and  perceptual  skill.  The  goal  of  this  study  was  to  see 
if  objective  tests  of  such  abilities,  which  have  already  been  shown  in  the  literature 
to  predict  surgical  performance,  also  might  predict  performance  on  the  ES3.  The 
subjects  of  the  study  were  34  medical  students  from  the  Albert  Einstein  College 
of  Medicine  and  4  otolaryngology  residents  from  Montefiore  Medical  Center. 

Results  of  this  study  revealed  several  correlations  among  these  previously 
validated  tests  and  performance  on  the  ES3  (Table  1).  We  have  shown,  through  a 
simple  regression  model,  that  scores  from  (1)  the  PicSOr  (Picture  Surface 
Orientation)  test,  a  validated  instrument  developed  to  identify  aptitude  for 
extracting  information  on  3-D  structures  from  2-D  images;  (2)  the  Cube 
Comparison  mental  rotation  test;  and  (3)  total  score  on  the  MIST-VR  system,  are 
strong  predictors  of  ES3  performance. 

PicSOr  scores  also  have  a  strong  correlation  with  the  ES3  hazard  score,  a 
figure  that  represents  the  number  of  errors  committed  during  a  simulated  surgical 
procedure.  Thus,  individual  students  who  have  difficulty  extracting  3-D 
information  from  2-D  images  appear  to  have  similar  difficulty  manipulating  a  3-D 
simulated  patient  on  a  standard  2-D  monitor. 

Similarly,  we  found  that  total  scores  from  the  MIST-VR  trainer  show  a 
significant  correlation  with  dissection  scores,  navigation  scores,  injection  scores, 
and  ES3  total  scores,  all  of  which  have  a  strong  dependence  on  hand-eye 
coordination.  It  can  be  concluded  then  that  the  ES3  provides  an  accurate 
assessment  of  the  technical  skills  that  it  proposes  to  measure,  thus  providing 
evidence  for  the  construct  validity  of  this  instrument. 
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Table  1.  Correlations  among  psychometric  and  ES3  simulator  scores 


ES3  scores 
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(CR) 
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Total  score 

Regression 
model  fit 

(ES3  trial 
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Hazard  score 

r  =  0.5 
(P  <  0.00) 

r=  0.29 
(P  <  0. 1) 

r=  0.27 
(P  <  0.14) 

r=  0.36 
(P  <  0.06) 

ES3  trial  score 

r  =  0.3 
(P  <  0.2) 

r  =  0.43 
(P  <  0.01) 

r  =  0.45 
(P  <  0.01) 

r  =  0.57 
(P  <  0.00) 

r  =  0.63 
(P  <  0.01) 

Significant  Pearson  correlation  coefficients  shown  in  boldface  type 


VR  to  OR 

A  third  study,  on  the  VR-to-OR  transfer  of  training,  addresses  in  a  direct 
manner  the  question  of  whether  training  on  the  ES3  will  improve  the  endoscopic 
sinus  surgery  performance  of  otolaryngology  residents  and  reduce  the  incidence 
of  surgical  errors.  This  study  requires  the  collection  of  resident  data  from  different 
institutions  in  East  Coast  and  West  Coast  programs  (Montefiore  Medical  Center, 
New  York  University  Medical  Center,  New  York  Eye  and  Ear  Infirmary, 
Columbia  University,  Yale  University,  the  U.S.  Naval  Medical  Center  in  San 
Diego,  and  Madigan  Army  Medical  Center).  Otolaryngology  residents  in  their 
junior  year  (e.g.,  PGY  2  and  3)  are  assigned  either  to  the  experimental  group 
(those  receiving  conventional  sinus  surgery  training  and  ES3  training  prior  to 
operating  on  a  patient)  or  a  control  group  (those  receiving  only  conventional  sinus 
surgery  training).  Individuals  in  the  two  groups  are  videotaped  during  one  of  their 
first  five  sinus  surgery  procedures;  these  tapes  then  are  scored  by  a  select  group  of 
trained  raters.  We  have  enrolled  eight  residents  from  Montefiore  Medical  Center, 
thus  far,  and  all  eight  of  them  have  completed  the  protocol.  Because  of  the  study’s 
complexity  and  the  requirement  involving  residents  in  a  particular  year  of  their 
residency,  this  multi-institutional  project  is  still  ongoing.  Results,  while  not  yet 
conclusive,  suggest  a  positive  transfer  of  training  from  the  simulation 
environment  to  initial  OR  procedures. 

Previous  work  by  Edmonds  during  the  development  of  the  ES3  demonstrated 
that  surgical  confidence  correlates  highly  with  dissection  skill,  suggesting  that 
confidence  comes  with  successful  dissection  experience.17  Simulation  training 
appears  to  positively  affect  initial  operating  room  performance,  as  judged  by 
senior  surgeons  rating  anonymous  videotapes  of  these  procedures,  with  the 
strongest  effects  being  for  instrument  manipulation  and  surgical  confidence.17 
Novice  trial  scores  on  the  ES3  have  been  shown  to  be  significant  predictors  of 
overall  surgical  competency.  Of  the  various  ES3  performance  measures, 
dissection  score  on  the  novice  model  was  found  to  be  the  strongest  predictor  of 
surgical  competency  in  the  OR.  Results  from  a  group  of  nine  residents  followed  a 
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consistent  trend  that  suggested  transfer  of  training  from  the  ES3  to  several 
components  of  surgical  competency. 


Web-based  data  repository 

A  unique  feature  of  this  study  is  the  creation  of  a  World  Wide  Web-based 
Intelligent  Infonnation  Interface  (WI3)  to  serve  as  a  repository  for  results 
obtained  from  remote  ES3  simulators  located  at  various  institutions.  The  primary 
objectives  of  the  WI3  are  to  provide  a  centralized  server  for  project  documents 
and  data,  and  an  Internet  site  for  analyzing  and  comparing  the  results  of  this 
project.  The  database  architecture  was  designed  in  such  a  way  that  it  can  serve  as 
a  template  for  archiving  and  accessing  results  from  other  surgical  simulation 
projects,  as  well.  The  four  major  components  of  this  architecture  are  (1)  the  ES3 
simulator  computer,  which  will  remain  isolated  from  direct  Internet  access  (to 
assure  confidentiality  of  trainee  scores);  (2)  the  proctoring  software,  which  runs 
on  a  PC  that  is  linked  directly  to  the  SGI  simulation  host  computer  via  a  local 
network  and  with  the  central  database  server  via  the  Internet;  (3)  the  central  SQL 
server,  which  is  housed  at  the  University  of  Washington  and  manages  database 
input  and  access  functions;  and  (4)  the  database  access  software  made  available  to 
the  general  public  and  the  project  investigators,  via  a  standard  Internet  service 
connection  and  Web  browser  interface.  The  WI3  design  is  not  tied  to  the  ES3 
specifically,  but  is  readily  generalizable,  extensible,  and  scalable  to  other  surgical 
(and  nonsurgical)  simulators. 

The  system  architecture  supports  three  types  of  simulator  trial  database 
access:  (1)  immediate  access  to  the  trainee’s  percentile  score  and  learning  curve, 
relative  to  appropriate  comparison  groups  (e.g.,  senior  residents),  is  available 
through  the  proctor’s  interface  during  the  training  trial  debriefings — this  same 
information  also  can  be  retrieved  after  the  fact  through  the  password-protected 
Web  site  server;  (2)  project  investigators  can  view  simple  descriptive  statistics 
and  data  graphics  for  user-selected  database  subsets  directly  from  the  Web  site; 
and  (3)  investigators  can  download  user-selected  data  subsets  for  further  statistical 
analysis  using  their  PC  and  standard  data  analysis  software,  such  as  Excel®  or 
SPSS®. 

In  addition  to  the  simulator  trial  data,  the  server  archives  and  provides  access 
to  various  project  documents  including  user  manuals,  protocol  descriptions,  data 
graphics,  and  drafts  of  working  papers.  The  server  also  supports  streaming  media 
content,  such  as  videotape  samples,  and  facilitates  automated  data  uploading  from 
the  proctor’s  computer.  Without  the  latter  capability,  the  acquisition  and 
integration  of  data  from  remote  ES3  sites  would  be  considerably  more  difficult 
and  more  prone  to  data  transfer  errors.  The  remote  ES3  systems  located  on 
various  campuses  are  essential  to  achieving  adequate  subject  numbers  in  a  field 
such  as  otolaryngology,  given  that  the  residency  programs  are  so  small.  In 
addition,  the  WI3  will  serve  as  a  prototype  for  the  creation  of  a  national  database, 
whereby  each  resident’s  surgical  skill  can  be  compared  to  that  of  an  appropriate 
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cohort,  as  well  as  to  established  performance  criteria.  No  such  repository  for 
surgical  skill  data  exists  to  date. 


Conclusion 

The  use  of  advanced  simulation  technology  has  contributed  to  an 
extraordinary  aviation  industry  safety  record.  Likewise,  simulation  technology 
and  methodology  can  be  used  to  improve  the  quality  of  surgical  training  for 
procedures  such  as  endoscopic  sinus  surgery  (ESS),  one  of  the  most  common 
procedures  performed  by  otolaryngologists  in  the  United  States — and  one  that 
poses  significant  risk  to  the  surrounding  anatomic  structures,  such  as  the  eye  and 
brain. 

We  have  established  a  working  group  that  capitalizes  on  the  strengths  of 
numerous  investigators  and  institutions  with  expertise  in  the  area  of 
otolaryngologic  training,  image-guided  surgery,  simulation  development,  metrics 
development,  and  outcome  analysis.  The  overall  goal  of  the  collaborative  effort  is 
improved  patient  safety  for  surgical  procedures,  using  highly  sophisticated 
surgical  simulators  to  improve  the  screening,  education,  training,  error  detection 
skills,  assessment,  and  certification  of  surgeons.  The  resulting  technical  skill 
improvements  will  render  safer  operative  procedures  with  fewer  medical  errors. 

We  anticipate  that  the  next  generation  of  ES3  will  generate  a  virtual  surgical 
environment  based  on  individual  patient  anatomy  data  (derived  from  the  patient’s 
own  CT  scan)  that  can  be  downloaded  into  the  simulator.  The  technology  will 
enable  the  surgeon  to  perform  a  “mission  rehearsal”  prior  to  the  actual  surgery, 
for  the  purpose  of  anticipating  risks  and  hazards,  and  annotate  them  with 
radiology  images  to  be  used  during  the  actual  surgery.  We  further  anticipate  that 
surgical  simulators  of  the  sort  described  in  this  paper  will  achieve  a  level  of 
performance  in  the  next  few  years  that  will  make  them  suitable  for  surgeon  testing 
and  certification. 
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